Evaluation Documentation #452
Conversation
… scoring.py and filled out the readme for metrics.
robertbartel
added
documentation
|
There's more to be done, but I'm opening this up for review just so we can get stuff in there - perfection is the enemy of progress and all that. |
|
Looking at this now. |
There was a problem hiding this comment.
This is great, @christophertubbs! I left a few comments, several of which are minor grammatical errors and others are optional suggestions.
python/lib/evaluations/README.md
Outdated
There was a problem hiding this comment.
Assuming this is what you are working on now.
| `Communicator` objects to help distribute information generated during evaluation. Next, you can | ||
| call the `ScoringScheme` as a function with prepared pairs, where to find "observed" values within | ||
| the pairs, where to find the "predicted" values within the pairs, and thresholds. This invocation | ||
| will yield a `MetricResults` object. |
There was a problem hiding this comment.
Is there a minimum viable example you could provider here? If its more than, say 5 lines, maybe it is not worth it. But it would nice to see all the concepts you described above in action. I think it is okay to let the reader imagine the data flowing through the system.
python/lib/metrics/README.md
Outdated
|
|
||
| Each `MetricResults` object contains the evaluated metrics performed on a singular set of data. | ||
| When providing the `Metric`s that are run to the `ScoringScheme`, a weight is passed along for | ||
| each metric and passing thresholds to the invocation will provide weights for each threshold. This |
There was a problem hiding this comment.
| each metric and passing thresholds to the invocation will provide weights for each threshold. This | |
| each metric. Passing thresholds to the invocation will provide weights for each threshold. This |
python/lib/metrics/README.md
Outdated
| Each `MetricResults` object contains the evaluated metrics performed on a singular set of data. | ||
| When providing the `Metric`s that are run to the `ScoringScheme`, a weight is passed along for | ||
| each metric and passing thresholds to the invocation will provide weights for each threshold. This | ||
| provides a basis for determine a hierarchy of importance for each metric and threshold. For an |
There was a problem hiding this comment.
| provides a basis for determine a hierarchy of importance for each metric and threshold. For an | |
| provides a basis to establish a hierarchy of importance for each metric and threshold. For an |
| This means that the `info` event will be triggered on each held `Communicator`, but only on those set to handle | ||
| messages of a verbosity of `LOUD` or greater, and to call the `write` event after doing so. | ||
|
|
||
| Say I have three communicators: |
There was a problem hiding this comment.
It would be super nice to see how these three example Communicators are created / configured.
|
|
||
| Templates are environment specific - one environment may have an important template while another might not, | ||
| but the templates are configurable, so more and more may be created as new use cases arise. Template Manager | ||
| constructs (such as the [FileTemplateManager](template.py)) provide all the means necessary to find out what is |
There was a problem hiding this comment.
| constructs (such as the [FileTemplateManager](template.py)) provide all the means necessary to find out what is | |
| constructs (such as the [FileTemplateManager](template.py)) provide all the means necessary to find out what templates are |
| The most important aspects defined by a `DataSourceSpecification` are: | ||
|
|
||
| 1. What fields to load | ||
| 2. What the data is measured in or how to find out |
There was a problem hiding this comment.
Are udunits supported? Or what is/are the expected unit type formats?
There was a problem hiding this comment.
I am now getting to the unit definitions section below. It might be worth linking that that section here. Thoughts?
| } | ||
| ``` | ||
|
|
||
| Retrieve data "path/to/file.json" in the style handled by the "JSON File" template |
There was a problem hiding this comment.
I am assuming the JSON File template is provided and always available? Likewise, the same can be said for the Instantaneous NWIS Streamflow template, correct?
| ``` | ||
|
|
||
| <a id="UnitDefinition"></a> | ||
| ## Unit Definition |
There was a problem hiding this comment.
Is there a list of supported units / a dependency (i.e. pint) that you can point to in this section?
Documentation for all aspects of evaluations